Repository navigation
Conversation
… under TLI Both-ends transactional pipeline: tx_writer (table->topic) and receive_batch_with_tx + sink upsert (topic->table), commit makes topic offset and table write atomic. TLI is induced via hot-row contention and absorbed by retry_tx; run stays exactly-once and loss-free.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #854 +/- ##
=======================================
Coverage 81.22% 81.22%
=======================================
Files 94 94
Lines 12111 12111
Branches 1184 1184
=======================================
Hits 9837 9837
+ Misses 1809 1808 -1
- Partials 465 466 +1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds new SLO workloads (sync-topic-tx / async-topic-tx) to validate an exactly-once transactional pipeline between YDB Topic and Table under chaos + deliberate TLI contention, integrating the new metric (topic_tx_tli) into SLO reporting and CI.
Changes:
- Implement transactional topic↔table producer/consumer job managers (sync + async) using
tx_writer/receive_batch_with_tx, plus runner/CLI wiring. - Add
topic_tx_tlimetric emission and expose it in metrics/threshold configs. - Document the scenario and update SLO README / workflow matrix / entrypoint ref-scoping for tx topics + tables.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tests/slo/TOPIC_TX_SCENARIO.md |
New design doc describing the transactional topic↔table SLO scenario and invariants. |
tests/slo/thresholds-topic.yaml |
Adds topic_tx_tli as a neutral (informational) threshold metric. |
tests/slo/src/runners/topic_runner.py |
Creates/drops tx sink+hot tables for *-topic-tx workloads; selects tx vs non-tx job managers. |
tests/slo/src/options.py |
Adds CLI options for sink/hot tables, hot-key count, and messages-per-tx. |
tests/slo/src/jobs/topic_tx_jobs.py |
New sync transactional workload implementation (topic→table + table→topic in one tx). |
tests/slo/src/jobs/topic_jobs.py |
Refactors validation to reuse _record() for both classic and tx workloads. |
tests/slo/src/jobs/async_topic_tx_jobs.py |
New async mirror of the transactional workload. |
tests/slo/src/jobs/async_topic_jobs.py |
Refactors validation to reuse _record() for both classic and tx workloads. |
tests/slo/src/core/metrics.py |
Adds inc_tli() and OTLP counter sdk.topic.tx.tli.total. |
tests/slo/README.md |
Documents new *-topic-tx workload labels and describes behavior/options. |
tests/slo/metrics-topic.yaml |
Adds PromQL query/series for topic_tx_tli. |
tests/slo/docker-entrypoint.sh |
Adds ref-scoped topic + sink/hot table naming for *-topic-tx workloads. |
.github/workflows/slo.yml |
Runs new sync-topic-tx / async-topic-tx workloads in the SLO workflow matrix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
AI Review Summary
Verdict: ✅ No critical issues found
Critical issues
No critical issues found.
Other findings
- Nit | High: Non-English word "штатный" in documentation and config comments —
TOPIC_TX_SCENARIO.md:12,thresholds-topic.yaml:33,metrics-topic.yaml:55 - Minor | Low: Table names interpolated into SQL via
.format()(standard for YDB, but worth noting as a code smell for maintainability) —topic_tx_jobs.py:56-58,topic_runner.py:59-67
Overall assessment: This is a well-designed PR. The transactional semantics are carefully handled:
seqnoadvances only after successful commit (no gap on abort)- The
committedlist is reset on retry vianonlocal(correct for retry_tx) - Sink UPSERT is keyed by
(writer_id, seqno)making re-processing idempotent - Hot rows are correctly seeded and
_bump_hotgracefully handles missing rows - The
_recordrefactor in parent classes is clean and preserves existing behavior - Thread safety is correct: sync uses
_expected_lock, async relies on single event loop - The
_validate→_recordsplit correctly separates message decoding from delivery tracking, allowing tx subclasses to call_recordonly after commit - Ref-scoping of topics and tables prevents cross-contamination between current/baseline containers
- The entrypoint correctly sanitizes ref names and scopes all resources
This review was generated automatically. Critical issues require attention; other findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.
| self.messages_per_tx = max(1, int(getattr(self.args, "messages_per_tx", 10))) | ||
| self.hot_keys = max(1, int(getattr(self.args, "tli_hot_keys", 4))) | ||
|
|
||
| self._sink_upsert_query = SINK_UPSERT_TEMPLATE.format(sink_table) |
There was a problem hiding this comment.
Severity: Minor
Confidence: Low
Table names are interpolated into SQL strings via .format(). This is the standard pattern in the existing SLO codebase (same approach in table_runner.py) and YDB doesn't support parameterized table names, so it's not exploitable here — especially since the inputs come from CLI args sanitized in docker-entrypoint.sh (via tr -c 'a-zA-Z0-9_' '_'). Just flagging for awareness since it's a new set of format-injected queries.
|
Analysis performed by claude, claude-opus-4-6. |
Remove the hot-row contention table, the tli_hot_keys/hot_table options, and the topic_tx_tli metric. The scenario keeps its point: a transactional topic write (tx_writer) and a transactional topic->sink-table consumer (receive_batch_with_tx + upsert), exactly-once under chaos. Single sink table.
read_availability was ~90% (async) not because reads failed but because a receive_batch_with_tx that blocks (idle reader, or the SDK transparently reconnecting under chaos) hit our wait_for timeout, which is not a ydb.Error so the tx retrier never retried it — and we wrongly recorded it as a failed read. Add metrics.cancel(): a timeout or an empty batch now balances the pending counter without a success/failure, so read_availability reflects the success rate of actual transactional reads (which the retrier drives ~100%). Real outages remain visible via write_availability / topic_delivered_rps.
🌋 SLO Test Results🔴 7 workload(s) tested — 2 workload(s) exceeded failure thresholds
Threshold violations: async-query:
async-topic-tx:
sync-topic:
Generated by ydb-slo-action |
|
Analysis performed by claude, claude-opus-4-6. |
| """Transactional topic <-> table pipeline (sync). | ||
|
|
||
| Producer: a ``tx_writer`` writes a batch of messages to the topic inside a | ||
| transaction (a transactional topic write). Consumer: | ||
| ``receive_batch_with_tx`` reads a batch while the same transaction UPSERTs | ||
| each message into a sink table keyed by ``(writer_id, seqno)``. The commit | ||
| advances the topic offset and persists the sink rows atomically -> a | ||
| rolled-back tx re-reads and re-UPSERTs the same keys (idempotent), so the | ||
| table receives every message exactly once, even under chaos. |
| def cancel(self, labels, start_time: float) -> None: | ||
| """Undo a ``start()`` for an operation that turned out to be a no-op | ||
| (e.g. a transactional reader that had no batch to process): balances the | ||
| pending counter without recording a success or a failure, so idle waits | ||
| do not count against availability.""" | ||
| return None |
There was a problem hiding this comment.
AI Review Summary
Verdict: ❌ Critical: 0; Major: 1
Critical and Major issues
- Major | Medium: No explicit timeouts anywhere on the transactional path (
retry_tx_sync/retry_tx_async,tx.execute,tx.commit,tx_writer.write); YDB's default gRPC deadline is 600 s, so one stuck RPC under chaos can block a writer/reader thread for up to 10 minutes — far beyond--time— whilerun_tests()joins these threads, stretching or stalling the whole SLO run —tests/slo/src/jobs/topic_tx_jobs.py:100
Other findings
- Minor | High: The
topic_duplicatesinvariant described in the scenario doc ("redelivery after a rolled-back tx") can never fire in the implemented flow, because_recordruns only after a successful commit, so rolled-back re-delivery is always recorded as a fresh delivery —tests/slo/TOPIC_TX_SCENARIO.md:73 - Minor | Medium:
topic_e2e_latencyis documented as "write-commit → read-commit latency" butwrite_ts_nsis captured at payload-encode time, before the producer transaction starts and before any retry backoff, so consumer-side e2e absorbs producer retry delays after chaos aborts —tests/slo/TOPIC_TX_SCENARIO.md:72 - Minor | Medium: The
TimeoutError→cancel()branch treats every receive timeout as an idle reader; a starved reader whose producers are all wedged in long (timeout-less) transactional writes also shows up as idle withread_availability~100%, hiding the stall from the availability signal —tests/slo/src/jobs/topic_tx_jobs.py:149
This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.
|
|
||
| ts = self.metrics.start((OP_TYPE_WRITE,)) | ||
| try: | ||
| pool.retry_tx_sync(callee, retry_settings=self.retry_settings) |
There was a problem hiding this comment.
Severity: Major
Confidence: Medium
The transactional path passes no timeout anywhere: retry_tx_sync(callee, retry_settings=self.retry_settings) runs a callee that calls tx_writer.write(...) (whose first write does a write_with_ack flush), and the pool wrapper then calls tx.commit(), both without any timeout; the consumer side (_run_topic_reads) likewise calls tx.execute for the sink UPSERT without settings. YDB's default gRPC deadline for calls without settings is DEFAULT_TIMEOUT = 600 s (ydb/connection.py), so under chaos a single RPC to a killed-but-TCP-alive node can block this writer thread for up to 10 minutes — far beyond --time — and run_tests() joins the job threads, so the whole run stalls with it. The plain topic jobs deliberately wrap every operation with write_timeout/read_timeout (see the "so the workload never silently hangs" comments in topic_jobs.py); this PR's tx jobs lost that protection.
Suggested fix: pass a bounded deadline into the transactional operations — e.g. build BaseRequestSettings(timeout=...) (from --write-timeout/--read-timeout) for tx.execute/tx.commit, and/or bound the whole retry attempt with asyncio.wait_for in the async manager the same way receive_batch_with_tx is bounded.
| | `topic_lost_errors` | forward gap in a producer's seqno (real loss) | **0** — hard-fails the run, under chaos | | ||
| | exactly-once into sink | sink keyed by `(writer_id, seqno)` — idempotent UPSERT | re-processing after a rollback creates no duplicate row | | ||
| | `topic_e2e_latency_p50/p99` | write-commit → read-commit latency | reported | | ||
| | `topic_duplicates` | redelivery after a rolled-back tx | informational (at-least-once at the app level) | |
There was a problem hiding this comment.
Severity: Minor
Confidence: High
This invariant row documents topic_duplicates as "redelivery after a rolled-back tx", but that signal is unreachable in the implemented flow: _record is called only after retry_tx_* returns (i.e. only after the commit succeeded), so a consumer tx that rolls back is never recorded at all — its messages are re-read and recorded once as fresh deliveries (seqno == exp), never as duplicates. The duplicates counter in this workload can only fire on a different mechanism: the producer re-committing an identical seqno range after a lost commit response.
Suggested fix: reword the row to describe the producer re-commit duplicate case, or note that consumer-side rollback redelivery is invisible to topic_duplicates by design.
| |--------|---------|-------------| | ||
| | `topic_lost_errors` | forward gap in a producer's seqno (real loss) | **0** — hard-fails the run, under chaos | | ||
| | exactly-once into sink | sink keyed by `(writer_id, seqno)` — idempotent UPSERT | re-processing after a rollback creates no duplicate row | | ||
| | `topic_e2e_latency_p50/p99` | write-commit → read-commit latency | reported | |
There was a problem hiding this comment.
Severity: Minor
Confidence: Medium
The table defines topic_e2e_latency_p50/p99 as "write-commit → read-commit latency", but the code captures write_ts_ns via time.monotonic_ns() inside encode_payload, which runs before the producer transaction starts (and before the limiter pacing loop completes for the rest of the batch). If a chaos abort forces retry_tx_* to re-run the callee, backoff (RetrySettings defaults grow to seconds per attempt over 10 retries) is absorbed into the consumer-side e2e measurement, so the metric drifts from its documented meaning and can inflate the gated topic_e2e_latency_p50 after aborts.
Suggested fix: either reword the doc to "encode → read-commit", or capture the write timestamp on the final attempt (e.g. right before tx.commit() / after the first successful write ack) so e2e measures what the doc claims.
| # the SDK transparently reconnecting under chaos, | ||
| # not a read failure. Don't count the op (real | ||
| # errors surface as exceptions the tx retries). | ||
| self.metrics.cancel((OP_TYPE_READ,), ts) |
There was a problem hiding this comment.
Severity: Minor
Confidence: Medium
Every receive timeout is classified as an idle no-op (cancel(), no success and no failure). That is correct for a genuinely empty topic, but it also covers the starvation case: if all producers are wedged in timeout-less transactional writes (see the missing-timeouts issue), readers sit here receiving nothing, and read_availability stays at ~100% with zero error counters — the availability signal cannot distinguish "no work to do" from "pipeline stalled". The plain topic reader intentionally counts a starved timeout as a read failure for exactly this reason (comment in topic_jobs.py).
Suggested fix: keep the no-op classification if desired, but make starvation visible another way — e.g. count consecutive empty timeouts per reader and emit them as a separate metric, or count a timeout as a failure after the reader has been idle for longer than the expected inter-batch interval.
What
Adds two SLO workloads —
sync-topic-tx/async-topic-tx— that validate the both-ends transactional topic ↔ table pipeline (exactly-once) under chaos and TLI.tx_writer(tx)writes a batch to the topic while the same tx does a read-modify-write on a shared hot row. Commit persists the topic write + table update atomically;seqnoadvances only on commit (no gap on abort).receive_batch_with_tx(tx)reads a batch while the same tx UPSERTs each message into a sink table keyed by(writer_id, seqno)and bumps a hot row. Commit advances the offset + writes rows atomically → a rolled-back tx re-reads and re-UPSERTs the same keys (idempotent, exactly-once).--tli-hot-keys) →ydb.Aborted(locks invalidated).retry_tx_*absorbs it (counted viaon_ydb_error_callback→topic_tx_tli, neutral). Even retry-exhaustion recovers with no loss.Reuses the existing topic writer/reader machinery, payload codec, metrics, and reconnect/recreate loops. Design doc:
tests/slo/TOPIC_TX_SCENARIO.md.Verified (local
/local, 30s each)black/flake8/py_compileclean. No CHANGELOG (internal tests/CI).